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ABSTRACT 

This paper describes a new approach to situation awareness that combines video sensor technology and synthetic vision 
technology in a unique fashion to create a hybrid vision system. Our implementation of the technology, called 
”SmartCam3D” (SC3D) has been flight tested by both NASA and the Department of Defense with excellent results. This 
paper details its development and flight test results. 

Windshields and windows add considerable weight and risk to vehicle design, and because of this, many future vehicles will 
employ a windowless cockpit design. This windowless cockpit design philosophy prompted us to look at what would be 
required to develop a system that provides crewmembers and operations personnel an appropriate level of situation 
awareness. The system created to date provides a real-time 
3D perspective display that can be used during all-weather and 
visibility conditions. While the advantages of a synthetic 
vision only system are considerable, the major disadvantage of 
such a system is that it displays the synthetic scene created 
using “static” data acquired by an aircraft or satellite at some 
point in the past. The SC3D system we are presenting in this 
paper is a hybrid synthetic vision system that fuses live video 
stream information with a computer generated synthetic scene. 

This hybrid system can display a dynamic, real-time scene of 
a region of interest, enriched by information from a synthetic 
environment system, see figure 1. 

The SC3D system has been flight tested on several X-38 flight 
tests performed over the last several years and on an ARMY 
Unmanned Aerial Vehicle (UAV) ground control station 
earlier this year. Additional testing using an assortment of 
UAV ground control stations and UAV simulators from the 
Army and Air Force will be conducted later this year. 

We are also identifying other NASA programs that would benefit from the use of this technology. 

Keywords: Synthetic vision, simulation, flight visualization, flight guidance, human factors, situation awareness, hybrid 
synthetic system, UAV visualization, and smart camera systems. 



Figure 1 : A Smart Camera System in action during Vehicle 131 R 
flight-testing. Here we see video that is augmented with 
synthetically created runways, buildings, and no-fly zones. We 
also included a HUD and labeled the Rosamond Dry Lake bed. 


SMART CAMERA SYSTEM CONCEPT 


The goal of the SC3D system is to create a system that provides crews and operations personnel with enhanced situation 
awareness (both for comfort and safety). The X38 program provided an initial testbed for the SC3D system. The X38 
program was a NASA effort investigating the requirements to develop a Crew Return Vehicle (CRV) for the International 
Space Station (ISS). The X38 vehicles employed a windowless cockpit design philosophy. As such, crewmembers onboard 
would have to rely on technology advancements to provide them with the visualization information required during all 
phases of their mission. The hybrid system developed augments a live sensor data stream (video in this case) with 
information from a synthetically created scene. The system provides operators visuals that continue to function under all 
weather and visibility conditions. In cases of night, poor weather or other limited visibility environments, the operators can 
utilize the computer-generated synthetic out the window scene. However, if video data is available, it can provide a view that 
includes transient objects that might not be present in the geographic database. Obstructions, hazards, landing areas, flight 
paths/corridors, airstrips, and other items can be easily created synthetically and inserted into the live video stream. These 
synthetically created elements will continue to function, regardless of the quality of video stream. Figure 2 is a depiction that 
shows the basic components and data flow for the SC3D system system. 



Figure 2: basic Components of the Smart camera System 


The notion of a hybrid synthetic vision system is simple. A video camera is mounted on the vehicle in such a manner as to 
provide the operator with a view of the vehicle flight path. Simultaneously, a 
computer creates a 3D representation of the current scene that the camera 
should be viewing. Doing this requires a camera bore-sight calibration 
procedure that has been developed and verified to co-align the real and 
simulated cameras views, see figure 2. Once this alignment is performed, the 
two streams of information (video and synthetic) are overlaid inside the 
computer and displayed in a Helmet Mounted Display (HMD) or standard 
display monitor. Figure 3 shows a sample frame from the system that has 
been developed. Crewmembers and operations personnel assigned to evaluate 
this display system have noted that the addition of a custom Head-Up Display 
(HUD) is of great benefit. For pilots of both full scale and remotely piloted 
aircraft, such a display system would provide a view of the environment, 
which includes live video and enhanced with outlines of the terrain, other 
aircraft, landing zones, targets, objects of importance, runway outlines, 
landing zone markers, building models, no-fly zones, checkpoint flags, 
landing obstruction markers, hazardous landing areas, and flight path 
corridors and a custom HUD. The currently implemented HUD includes: an altitude tape, a compass heading indicator, a 



Figure 3: Combined video, synthetic scene, and 
overlavs. 


pitch ladder, a bank angle indicator, control surface command and position indicators, and miscellaneous flight data. 
Furthermore, because the synthetically created objects are generated from digital data, they are not subject to the limitations 
of visibility inherent to video. While darkness, terrain occlusion, smoke, fog, and haze all impact the video, the synthetically 
generated items will be unobstructed. 


PLATFORM REQUIREMENTS 

The SC3D system developed during our initial phase was 
deployed to the X-38 Remote Cockpit Van (RCV), see reference 
3 for detailed description of the RCV, but could be deployed on 
any platform that meets the SC3D system fundamental 
requirements. The fundamental requirements include a real- 
time video stream, position & attitude data for the aircraft, and a 
graphical information system (GIS) database. This specific 
vehicle flight data required include: latitude, longitude, altitude, 
pitch, heading and roll, which we call our 6 degree of freedom 
(6-DOF) data set. Additionally, if the camera is mounted on a 
gimbal and has zoom capability, then gimbal angles and zoom 
setting are also required in the data stream. Realistic 
performance is achieved if this data is received at a data rate that 
is greater than 5 Hz. The 6-DOF data allows the software to 
synthesize a real-time 3D scene to match the camera’s view. 


The term video, as it is used in this discussion, refers to visual 
wavelength light but could include any wavelength signal from 
visual wavelength, to Near Infrared (NIR), Long Wave Infrared 
(LWIR), Forward-Looking Infrared (FLIR), etc. Anything that 
produces an RS-170 video or a stream of digital images that can 
be move into computer memory quickly can be utilized as the 
input video stream into the SC3D system. 

The computer inserts information that details the location of landing zones, no-fly zones, runways, obstructions, buildings, 
topography and other geographic data. Anything with known geographic coordinates can be included in the scene. The 
synthetic image created exactly matches the theoretical camera view, and is overlaid on the video in real time. Base on our 
experience, modern laptops can generate these overlays at 5 to 15 frames per second. Figures 4 and 5 show examples frames 
taken from the first flight test of the X-38 13 1R vehicle. 

FLIGHT TESTING 

The X-38 vehicle was equipped with an Embedded GPS/INS system (EGI). The EGI provides vehicle position and attitude 
information to the flight control system and to the telemetry system that downlinks the data to the engineers performing flight 
following activities on the ground stations. The onboard telemetry system transmits all of the vehicle parameters, including 
the vehicles position (latitude, longitude, and altitude) and orientation (pitch, heading and roll) angles and are available for 
view by engineers and operations personnel in the X38 Mission Control Center at JSC, at the Spectrum Analysis Facility 
(SAF) at the Dryden Flight Research Center (DFRC), and in the Remote Cockpit Van in the field. Additionally, the X-38 
13 1R vehicle was equipped with a camera mounted on the underside of the nosecone. The video from this camera is also 
transmitted to the ground. Prior to a flight test, ground crews activate the camera and perform a brief calibration procedure to 
determine the exact field of view and pointing angles (pitch, reading and roll offsets) for the onboard camera. This bore-sight 
calibration data is used by the SC3D system to create an identical viewpoint for the synthetic scene as the one being seen by 
the “real” camera. 

The X-38 13 1R vehicle made three successful free flights at the DFRC. The SC3D system was preloaded with a 3-D 
topographical model of the flight test area, synthetically created 3D entities, maps/aeronautical charts for the area, and a 
custom built HUD. The 3-D synthetic entities created for the flight test include: a geographic model of the base, prominent 
landmarks, and reference points of interest to the mission. Runways, large hangars, towers, and buildings in the geographic 



Figure 4: The Smart Camera uses the 3D position and attitude of the 
vehicle and thus provides situation awareness, regardless of vehicle 
attitude. Here the vehicle has rolled almost inverted, but the Smart 
Camera display still correctly marks the location of objects in the scene, 
like the lakebed. 


model are displayed by the SC3D system and used as landmarks by the crew. Once a landing site has been selected, it is 
entered into the database and can be used as an aim point during final approach and flare flight phases. 

Results from this system have been very encouraging and have emphasized some of the less obvious benefits of the SC3D 
system. During one of the flights a rapid descent caused ice to form on the vehicle’s camera window, partially obscuring the 
cameras view, see figure 5. Astronauts in the remote cockpit, as seen in figure 6, were able to continue to rely on the display 
even when the camera had partially failed because of the presence of the synthetic vision overlays still correctly showed the 
location of landmarks and the landing zone. Free flight 1 also afforded the crew an opportunity to evaluate the display during 
a 360 degree roll. Because the SC3D system utilizes the real-time vehicle attitude information to synthesize its scene, the 
display continued to work very well, even at extreme attitudes (upside down at a pitch angle of -70 degrees). 
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Figure 5: Smart Camera provides additional safety against 
camera failure. Here the camera window has begun to ice 
over on the left side of the screen , but the crew is still seeing 
the synthetic elements of the 3-dimensional scene. 



Figure 6: Astronauts in the X-38 reimte cockpit using 
Smart Camera situation awareness teclmology. 


Video and flight data from both a helicopter and a glider have been evaluated in a post-processing regime with equally good 
results, and some significant discoveries emerged. During testing with an Army UH-1 helicopter it was discovered that gyro 
lag in older mechanical inertial navigation unit caused a perceptible lag in the angle-dependent elements of the overlay. 
While perceptible, this problem was mostly aesthetic. Data collected from a glider equipped with an air data system that 
included a barometric altimeter confirmed that barometric altitude measurements are several times more accurate than GPS 
altitude measurements. 



Figure 7 : A frame from an ARMY UAV flight display shows the Tactical 
Automated Landing System (TALS) Acquisition Window as a 3D corridor. 


Real time flight-testing on a United States Army Tactical 
Unmanned Aerial Vehicle (UAV) demonstrated both the 
feasibility of integrating the SC3D system with currently 
existing UAV ground control stations and the utility of 
SC3D system enhanced the situation awareness of the 
crew and ground control engineers. The flight data 
showed a good basic match between the synthetic and 
video scenes with some predictable exceptions. Engineers 
for the ground control station of the UAV alerted us to 
some inherent problems that we might encounter because 
of the UAV model that we would be using. We did 
encounter those problems and found although distracting 
at times, did not detract from the usefulness of the SC3D 
system for UAV operations. 


DEVELOPMENT PLANS 


An early version of the SC3D system demonstrated the usefulness of such a system using various “hardware mixer” boxes to 
perform the overlay operation. While this was useful in demonstrating the basic concept, the achievable accuracy, large form 
factor, and high cost made this option less desirable than a software only solution that would run on a desktop PC. NASA’s 
Integrated Video program started in 2001 and aims to achieve the software overlay capability, add support for multiple 
cameras, and incorporate various display technologies (i.e. a Helmet Mounted Display) into the SC3D system system. Flight 
testing will proceed along with development. More extensive data collection with fixed wing aircraft, the remote cockpit, 
and various simulators will be conducted this coming year. 

ENHANCED UAV DISPLAY 

Unmanned Aerial Vehicles (UAVs) 
often have on-board cameras, 
which transmit live video during 
flight to the control stations on the 
ground. As such, they can benefit 
significantly from the SC3D system 
technology. There are several 
improvements in the effectiveness 
of UAVs for combat applications 
for this technology. 

• The SC3D system makes 
it easier for operators to 
identify known friendly 
force locations in the 
video scene. 

• The synthetic vision 
system is based on 
VisualFlight, which is 
already compatible with 
most NIMA data formats 
including DTED, ADRG, 

CADRG, CIB, etc., so the 
necessary geographic data is already abundantly available. 

• The system aids operators in avoiding anti-aircraft weapons and small arms fire from known enemy locations - these 
can be shown as threat domes and are more easily avoided by the operator. 

• Safer vehicle operation at night and in limited visibility situations (sandstorms, dust, smoke, etc.). The SC3D 
system provides a synthetic vision component that is not affected by visibility conditions. 

• Finally, users will better utilize UAV video data because landmarks and reference points are clearly identified in the 
scene. 

STUDENT PILOT FLIGHT REVIEW SYSTEM 

The SC3D system could become an important tool for the general aviation industry, as a student pilot flight review system. 
In this application a training aircraft would be outfitted with the Smart Camera System including video camera, recorder, and 
computer laptop. The annotated video recorded during the flight would be reviewed with the student during debriefing. 

The student could also take a copy of the tape with him or her for later review. This would provide considerable benefit to 
the student by leveraging every hour of airtime into 2 or more hours of learning experience. Any available digital avionics 
information deemed critical to the lesson (airspeed, descent rate, pitch angle) would be displayed on the 2-D overlay to aid 
post-flight review, see figure 9. 
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Figure 8: Video camera frame taken in the downrange portion of the flight shows yellow no fly 
markers over courtesy no-loiter zones. Imaging target locations are indicated in black, as is the 
SmartCam3D HUD. 



RECONNAISSANCE, MISSION PLANNING, AND MISSION OPERATIONS: 


As military special operations, RECON, and other forces 
operate in geographically unfamiliar territory, the SC3D 
system will provide them with instant terrain familiarity. 
A man-portable SC3D system can be engineered to 
support field operations and provide operatives an 
increased level of situation awareness. Figure 10 shows 
an example of such a system in operation. Tags are used 
to indicate the location of significant landmarks, 
structures, and the positions of known friendly and 
enemy forces. The location of air strike or artillery aim- 
point coordinates can be viewed in three dimensions in 
advance of the strike, permitting the observer to confirm 
that the impact point is correct. An interactive system 
will permit the user to interrogate the database through 
the camera to obtain information about objects, which are 
in the scene and in the 3-D database. 
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Figure 9: Student Pilot night review system shows the FA A non-precision 
approach surface. 


CONCLUSION 

The NASA Smart Camera System has been developed to 
substantially enhance the information density and usefulness 
of video imagery. The SC3D system has been implemented 
using inexpensive off-the-shelf hardware and software, and 
it’s utility has been demonstrated during flight tests of the 
X-38 vehicle, flight test of an Army UAV, UH-1 flight 
testing, glider testing, and in the remote cockpit van (RCV). 
Future development will make the technology even simpler 
and more useful. A multiple camera feature, that will 
provide users with a live data 360 by 360 degree field of 
view of the area, will be incorporated this coming year. 
Additional testing will also be performed to determine if the 
technology can be used in several other application areas. 
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Figure 10: A portable smart targeting system may help Special Forces 
personnel in unfamiliar terrain make reliable targeting decisions. 
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Output stream of camera 
video combined with 
real-time synthetic vision 
imagery. 


Computer combines the video 
stream with the synthetically 
generated scene. 


The computer will also insert 
any additional overlays / 
information that may be 
necessary. 





Smart Camera Laptop PC 


Live Video (RS-170, 
Digital Video, or other) 
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UAV Ground Control Station 
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